Fix #64: Render copilot conversations including Markdown tables in SWT web browser widget#330
Fix #64: Render copilot conversations including Markdown tables in SWT web browser widget#330travkin79 wants to merge 9 commits into
Conversation
728eb45 to
6e4f73e
Compare
485efc7 to
8734279
Compare
|
@travkin79 This is a big change. But from the screenshot, the modern ui rendering result looks charming. @iloveeclipse What do you think of this? I somehow towards the web tech but need to make sure this works fine in different platform, I'm not worry for Mac and Windows, but for Linux, I'm not sure. |
|
I haven't tried this patch on my RHEL 9.6 yet (I'm busy with other tasks), so just few questions:
... May be more. In general, if that is going to be added, a fallback would be nice because each platform has its own quirks with browser and/or OS (Linux & Mac: webkit but different UI toolkits, Windows: Edge or IE for ancient systems). But that would mean to keep double implementation & tests for some time. |
|
Hi @jdneo and @iloveeclipse, Thank you for your feedback. @jdneo, I'm glad you like the new look. @iloveeclipse, I'm currently testing all the features, there are many of them in the chat view. I will also check the things you mentioned. One thing making the migration a little easier is: we're only changing the conversation rendering (user and copilot turns), all SWT widgets for the user input shown below the conversation and also the tool bar remain SWT widgets and they work as before. The buttons rendered in the conversation are the ones that change and need proper attention while testing. At the moment I'm testing on Windows 11 and there are still some minor issues I'm fixing step by step. I could also test on MacOS Tahoe (Intel) and on RHEL 9.6. I think, testing all the migrated features is the main effort here. We could add the new preference for switching between old and new rendering to the copilot preferences UI in case we want to maintain both rendering implementations for a while. |
Yes we can have a setting to progressively roll out this feature. I haven't deeply looked into the code, just having a general design question regard the implementation: For the frontend implementation, how is the current state management looks like? Is it complex? I'm thinking if it is necessary to introduce things like React and Redux. |
The state management is kept very simple, only on Java side. Adding JavaScript frameworks like React, Redux, Vue.js, Svelte, etc. is just not necessary. I tried making the implementation as simple as possible, simplifying debugging and code maintenance. My goal was to keep as much existing code unchanged as possible and leaving the domain knowledge in Java. The state management is kept completely in Java code (mainly in |
4f22cc9 to
21502ac
Compare
|
Hello @iloveeclipse, Thank you for your hints and questions. Here are my answers.
Yes, everything renders fine. I found a minor Edge/WebKit difference when auto-scrolling to the bottom while copilot is streaming his response and fixed it already.
I've added a preference for switching browser-based conversation rendering on or off. The default is NOT using the browser-based rendering. I'm still testing and working on code clean-up tasks. But it seems. there are only minor issues left. |
21502ac to
ccc0b1c
Compare
|
Another update for this effort, in the near future, we may start to integrate copilot cli into the plugin, and the CLS side payload of the copilot cli is hugely different from the current local agent. It is more like the ACP style. I will do some investigation and see how to manage the session status and reflect to the ui. |
|
Thank you for the hint, @jdneo. In best case, we'd hide the details of reading the payload or sending requests behind service classes (accessible through |
|
Hi @jdneo,
I guess, the copilot CLI sessions are different from GitHub Copilot Desktop App sessions, are they? Are you planning to integrate those in Copilot for Eclipse, too (or the other way around, integrate Copilot for Eclipse sessions in Copilot CLI and/or GitHub Copilot Desktop)? |
ca360b8 to
9992a3f
Compare
…icrosoft#64) - Introduce SWT-browser-based widget for rendering conversations in copilot chat view - Introduce common interface for the existing StyledText-based conversation rendering and the new browser-based conversation rendering - Introduce commonmark Markdown parser and HTML renderer with extensions to properly render GitHub-Flavoured Markdown tables (and task item lists), i.e. fix issue microsoft#64 - Introduce preference for switching between StyledText-based SWT rendering and browser-based HTML rendering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wrap the chatViewSideEffect disposal in ensureRealm() to guarantee it runs on the correct data binding Realm thread, preventing potential InvalidThreadAccessException when unbinding from a non-UI thread. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace preference-based theme detection with the proper IThemeEngine service API. The previous approach read the theme ID from preferences which could be stale or unavailable. The new approach queries the active theme directly via IThemeEngine.getActiveTheme(). We're using reflection in order not to introduce a dependency to the SWT themes bundle and to avoid discouraged access warnings. Read actual widget background color as fallback in case theme support is not available. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Cover UiUtils.isDarkTheme() with mocked PlatformUI/Platform statics: WITH e4 CSS theming and WITHOUT theming - theming-unavailable detection samples the widget-background luminance. Add isDark(RGB) BT.601 luma tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ft#64) - Add 'Compare Chat View Rendering Alternatives' Copilot menu entry - Opens both rendering alternatives in one dialog, side by side, for visual comparison Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Both the browser-based (HTML) and the seasoned StyledText renderers ship behind the useBrowserBasedChatRenderer preference, so both need probe coverage. Select the renderer per launch instead of hardcoding it. - ProbeRunner: read -Dprobe.renderer (browser | styledtext) and set USE_BROWSER_BASED_CHAT_RENDERER accordingly. Default styledtext matches the production default, so plain runs still exercise the SWT widget tree. - chat-send-receive-001.json: restored to the strong StyledText widget-tree assertions (user-turn, copilot-turn, model-info-label) for the default renderer. - chat-send-receive-browser-001.json (new): browser-renderer variant that asserts the Browser widget exists, then uses sleep + screenshot because the HTML DOM is opaque to SWTBot widget locators. - REFERENCE.md: document the -Dprobe.renderer property and the two-probe workflow for covering both renderers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
9992a3f to
a4e17d3
Compare
|
Hello @jdneo and @iloveeclipse, I used a separate target definition and launch configs for testing which are not part of this PR, but are placed into a separate branch adding only one commit with these files. @jdneo, do you think, you (the team) could review and merge this PR some time soon? Otherwise, we might run into merge conflicts with the main branch, especially when major changes and code re-structuring are coming. |
There was a problem hiding this comment.
Pull request overview
This PR introduces an optional browser-based conversation renderer for the Copilot Chat view to correctly render GitHub-flavoured Markdown features (notably tables) by converting Markdown to HTML with commonmark and rendering it in an SWT Browser, while keeping the existing StyledText-based renderer as the default behind a preference toggle.
Changes:
- Add a new
BrowserConversationWidgetpipeline (HTML template + CSS + Java↔JS bridge) and a sharedIConversationWidgetabstraction to support switching renderers. - Centralize shared logic (theme detection, avatar/name resolution, thinking-title parsing, error message normalization, quota actions) to prevent drift between renderers.
- Add unit/plugin/SWTBot probe coverage to exercise the new rendering path and preference-driven renderer selection.
Reviewed changes
Copilot reviewed 56 out of 60 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| launch/Run Checkstyle in Copilot for Eclipse.launch | Adds an Eclipse launch config for running checkstyle:check. |
| launch/Build and test Copilot for Eclipse.launch | Adds an Eclipse launch config for clean verify. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/utils/UiUtils.java | Refactors dark-theme detection (reflective e4 theme engine + background luminance fallback). |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/preferences/messages.properties | Adds preference strings for selecting the chat renderer. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/preferences/Messages.java | Adds NLS keys for the new renderer preference strings. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/preferences/CopilotPreferenceInitializer.java | Registers default value for the new renderer preference. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/preferences/ChatPreferencesPage.java | Adds the renderer-selection checkbox to the preferences UI. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/UserTurnWidget.java | Uses AvatarService for consistent user display-name resolution. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/ThinkingTurnWidget.java | Uses shared ThinkingTitles helpers for title params/persistence. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/ThinkingTitles.java | New shared helper for thinking-title extraction/persistence rules. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/ThinkingBlock.java | Reuses shared thinking-title pattern via ThinkingTitles. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/SvgIcons.java | New SVG icon registry for inline DOM rendering (browser renderer). |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/StyledTextConversationWidget.java | New adapter implementing IConversationWidget over the existing ChatContentViewer. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/services/UserPreferenceService.java | Ensures unbinding side effects happen in the correct realm. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/services/AvatarService.java | Adds avatar/name helpers + PNG data-URI support for browser embedding. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/services/AgentToolService.java | Routes tool confirmations through IConversationWidget instead of SWT-only widgets. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/QuotaActions.java | Adds QuotaPlanContext and helpers shared by both renderers. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/IConversationWidget.java | New renderer abstraction interface for the chat conversation area. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/CopilotTurnWidget.java | Uses AvatarService for consistent Copilot display-name resolution. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/ConversationHtmlBlockFactory.java | New HTML block factory (Markdown→HTML via commonmark + DOM fragments). |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/ChatView.java | Instantiates the selected renderer, supports live switching, and centralizes todo updates. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/ChatErrorMessages.java | New normalization of user-facing error messages shared by both renderers. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/ChatContentViewer.java | Delegates error normalization and removes todo-update logic (moved to view). |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/BrowserConversationJavaJsBridge.java | New Java↔JS bridge for DOM operations + browser callbacks. |
| com.microsoft.copilot.eclipse.ui/src/com/microsoft/copilot/eclipse/ui/chat/BaseTurnWidget.java | Centralizes quota-plan decomposition via QuotaPlanContext. |
| com.microsoft.copilot.eclipse.ui/resources/html/icons/warning.svg | Adds inline SVG used by browser warning blocks. |
| com.microsoft.copilot.eclipse.ui/resources/html/icons/thinking-bulb.svg | Adds inline SVG used for sealed thinking blocks. |
| com.microsoft.copilot.eclipse.ui/resources/html/icons/terminal.svg | Adds inline SVG used for tool confirmation blocks. |
| com.microsoft.copilot.eclipse.ui/resources/html/icons/pull-request.svg | Adds inline SVG used for agent-message PR cards. |
| com.microsoft.copilot.eclipse.ui/resources/html/css/chat-browser-light.css | Adds light-theme CSS for browser chat rendering. |
| com.microsoft.copilot.eclipse.ui/resources/html/css/chat-browser-dark.css | Adds dark-theme CSS for browser chat rendering. |
| com.microsoft.copilot.eclipse.ui/resources/html/css/chat-browser-base.css | Adds base structural CSS for the browser chat DOM. |
| com.microsoft.copilot.eclipse.ui/resources/html/chat-view.html | Adds HTML+JS template (DOM ops, auto-scroll, button event delegation). |
| com.microsoft.copilot.eclipse.ui/META-INF/MANIFEST.MF | Adds commonmark bundle dependencies for Markdown→HTML rendering. |
| com.microsoft.copilot.eclipse.ui/build.properties | Ensures resources/ (HTML/CSS/icons) are packaged. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/utils/UiUtilsThemeTest.java | Adds tests for dark-theme detection logic and luminance fallback. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/chat/SvgIconsTest.java | Adds unit tests for SVG attribute injection/merging. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/chat/services/AvatarServiceTest.java | Adds coverage for new avatar/name + data-URI behavior. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/chat/ChatErrorMessagesTest.java | Adds tests for error-message normalization. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/chat/BrowserConversationWidgetTest.java | Adds basic creation/disposal integration test for browser widget. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/chat/BrowserConversationWidgetBehaviorTest.java | Adds behavior tests for browser widget turn lifecycle/tool confirmation/restore logic. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/chat/BrowserConversationWidgetAvatarTest.java | Adds tests ensuring browser widget delegates avatar/name lookup to AvatarService. |
| com.microsoft.copilot.eclipse.ui.test/src/com/microsoft/copilot/eclipse/ui/chat/BrowserConversationJavaJsBridgeTests.java | Adds pure unit tests for emitted JS strings and escaping. |
| com.microsoft.copilot.eclipse.ui.test/plugin.xml | Adds a test-only command/handler/menu entry for visual renderer comparison. |
| com.microsoft.copilot.eclipse.ui.test/META-INF/MANIFEST.MF | Adds theme bundle dependency needed by theme-detection tests. |
| com.microsoft.copilot.eclipse.ui.jobs/build.properties | Removes explicit output directory (aligns build configuration). |
| com.microsoft.copilot.eclipse.ui.jobs/.settings/org.eclipse.jdt.core.prefs | Tightens forbidden reference severity to error. |
| com.microsoft.copilot.eclipse.terminal.api/build.properties | Removes explicit output directory (aligns build configuration). |
| com.microsoft.copilot.eclipse.swtbot.test/src/com/microsoft/copilot/eclipse/swtbot/test/probe/ProbeRunner.java | Adds -Dprobe.renderer support to run probes against either renderer. |
| com.microsoft.copilot.eclipse.swtbot.test/probe-scripts/chat-send-receive-browser-001.json | Adds a probe script for the browser renderer flow. |
| com.microsoft.copilot.eclipse.core/src/com/microsoft/copilot/eclipse/core/utils/DataUriUtils.java | Adds utility for base64 data: URI encoding. |
| com.microsoft.copilot.eclipse.core/src/com/microsoft/copilot/eclipse/core/utils/BundleUtils.java | Adds resource-loading helpers (bytes/string/data-URI) for OSGi bundles. |
| com.microsoft.copilot.eclipse.core/src/com/microsoft/copilot/eclipse/core/Constants.java | Adds preference key for browser-based chat renderer. |
| com.microsoft.copilot.eclipse.core.test/src/com/microsoft/copilot/eclipse/core/utils/DataUriUtilsTests.java | Adds unit tests for DataUriUtils. |
| com.microsoft.copilot.eclipse.core.test/src/com/microsoft/copilot/eclipse/core/utils/BundleUtilsTests.java | Adds unit tests for BundleUtils. |
| base.target | Adds Orbit IUs for commonmark and related dependencies. |
| .github/skills/ui-action/REFERENCE.md | Documents how to select the chat renderer for SWTBot probe runs. |
| if (userTurn.getMessage() == null | ||
| || StringUtils.isNotBlank(userTurn.getMessage().getText())) { | ||
| BaseTurnWidget userTurnWidget = | ||
| viewer.getLatestOrCreateNewTurnWidget(turn.getTurnId(), false, true); | ||
| userTurnWidget.appendMessage(userTurn.getMessage().getText()); | ||
| userTurnWidget.flushMessageBuffer(); | ||
| } |
There was a problem hiding this comment.
That should be improved, but that's original code from ChatView that I extracted, but left unchanged
| private void executeScriptForResult(String expression, Consumer<Boolean> onResult) { | ||
| if (browser.isDisposed()) { | ||
| onResult.accept(false); | ||
| return; | ||
| } | ||
| Display.getDefault().asyncExec(() -> { | ||
| if (browser.isDisposed()) { |
| private boolean validToolConfirmInvokeParams(String conversationId, String turnId) { | ||
| if (boundChatView == null) { | ||
| return false; | ||
| } | ||
|
|
Overview
This PR fixes rendering Markdown tables (GitHub-flavoured Markdown syntax) by introducing a new browser-based rendering approach based on an Eclipse-internal SWT web browser and HTML code created by converting Markdown to HTML using commonmark.
The new rendering approach is encapsulated in a
BrowserConversationWidget. It only covers conversations in the chat view. The other SWT widgets in the chat view stay unchanged, i.e. the input field, send button, toolbar, etc. The existing copilot conversation rendering is kept working and is still active by default. A new preference allows switching between both approaches. The currently used rendering approach was extracted toStyledTextConversationWidget. Both classes implement the new common interfaceIConversationWidget. After extensive review and testing, I'd suggest eventually removingStyledTextConversationWidgetand the corresponding new preference for switching between theStyledText-based rendering and the new browser-based rendering.The
BrowserConversationWidgetrenders HTMLDIVcode blocks representing existing SWT widgets like user / copilot turns, copilot replies, thinking, tool calls, tool confirmation UI, etc. Instead of re-rendering the whole conversation including long copilot turn HTML blocks, theBrowserConversationWidgetonly inserts new or modifies existing HTML code blocks / attributes to efficiently update the rendered HTML, e.g. while copilot is streaming a reply or thinking text. The look in theBrowserConversationWidgetis made very similar to the existingStyledText-based chat view implementation, i.e. it uses the same colors (but in separate CSS files since the e4 CSS files could not be re-used directly), same interaction features and behavior, e.g. automatically collapses a thinking block as soon as thinking finishes, etc. The JavaScript code in the HTML rendering template is kept very simple, the domain knowledge is kept in Java code.Switching between both rendering approaches can be done using a new Copilot preference:

Interaction between
BrowserConversationWidgetand the JavaScript code in the web browser:This PR includes a few additional fixes. I extracted them into separate commits in order to make reviews easier. Tests are also added in separate commits.
Testing and comparing the rendering approaches
For convenience while testing (in a separate git branch, not part of this PR), I also added a new target definition file and launch configs for running the Eclipse IDE with all bundles necessary for testing the two rendering approaches with and without themes support.
target-2025-09.targetas active target platform, firstlaunch/Copilot4e with Themes.launchlaunch/Copilot4e without Themes.launchlaunch/Copilot4e (compare chat view rendering approaches).launchfor side-by-side comparison (otherwise, i.e. without thecom.microsoft.copilot.eclipse.ui.testbundle, the corresponding menu entry is not available, details below)I added a Copilot menu entry for opening the two rendering alternatives side by side for visual comparison (only available in

com.microsoft.copilot.eclipse.ui.testbundle).Most elements visible in a conversation widget (user turns, copilot turns, tool calls, thinking, sub-agent calls, etc. in various statuses) are rendered here with both rendering approaches side by side using some artificial data. You can switch between dark and light theme, too. This way, you can easily check if the conversation details are rendered in the browser-based widget as you would expect them to be rendered.
Major design decisions
StyledText-based widgets that don't allow rendering tables in grid layout with proper bordersStyledText-based renderer is the default.small HTML
DIVcode blocks for e.g. reply, thinking, or tool call text blocks or update a few HTML attributes,e.g. to update a tool call status
Performance
I didn't compare the rendering performance of both approaches, but I implemented the browser-based rendering in such a way that only tiny DOM manipulations are needed during conversation widget content updates (e.g. adding single words while copilot is in thinking state). While testing with exemplary requests to copilot I didn't see any observable slow-downs.
Suggested follow-up tasks for separate PRs
EditAgentRoundData(thinking -> reply -> toolCalls). Applies to both theStyledText-based and the browser-based conversation rendering.com.microsoft.copilot.eclipse.ui.testbundle (and maybe other test bundles, too) a fragment bundle instead of a separate OSGi bundle (use Fragment-Host entry in MANIFEST.MF). This way we can avoid making production code package-private methods public, just for testing them from a test bundle. The following methods, for example, should all be package-private, not public: inBrowserConversationJavaJsBridge:escapeForJs,insertBlockScript,insertBlockBeforeScript,replaceBlockScript,removeBlockScript,collapseThinkingBlockScript,updateThinkingBodyTextScript,updateThinkingBlockTitleScriptandUiUtils.isDark.ImageRegistryfor all images (the registry takes care of image disposal).